{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7c488a29",
   "metadata": {
    "id": "CDFEB8D3A1A341D39E84BE2F1F955D28",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "# Chapter-2 大模型使用  \n",
    "\n",
    "本节学习大纲  \n",
    "\n",
    "- 掌握通过 API 调用云端大模型的基本方法。  \n",
    "- 掌握使用 transformers 库在本地运行大模型的基本流程。  \n",
    "- 了解 vLLM 等高性能推理框架的用途和基本使用方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a3eab4e",
   "metadata": {
    "id": "B935764159934AD396C6554463D75E42",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "上一节课我们主要解了什么是大模型，以及在哪里可以找到它们，比如 Hugging Face Hub 以及 ModelScope。本节的主要目标是“动手”，从理论走向实践，真正地把大模型用起来。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a0cef57",
   "metadata": {
    "id": "03A5E2B4967941ACA9A34828D21408EE",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [],
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "使用大模型主要有两种路径：  \n",
    "\n",
    "1. 云端API调用：像使用一个在线服务一样，我们发送请求，远端的服务器（云）返回结果。这种方式简单、方便，对我们自己的电脑配置要求不高。  \n",
    "\n",
    "2. 本地化部署：把大模型下载到我们自己的电脑（服务器）上并运行起来。这种方式给了我们最大的控制权和隐私保障，但对硬件（尤其是显卡）有一定要求。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8826055b",
   "metadata": {
    "id": "E915B3FE10E54520A28A022CE78BA48F",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 2.1 云端大模型调用：最便捷的路径  \n",
    "\n",
    "这是使用大模型最直接的方式。我们不需要关心硬件配置、模型加载等复杂问题，只需要获得一个许可（API Key），然后按照服务商提供的地址和格式发送请求即可。  \n",
    "\n",
    "我们将以[硅基流动（SiliconFlow）](https://cloud.siliconflow.cn/i/ybUFvmqK)平台为例，它提供了与 OpenAI API 完全兼容的接口，我们可以直接使用 OpenAI 中的 Python 库调用模型。  \n",
    "\n",
    "> 注：虽然我们使用 `openai` 库的方式来调用云端大模型，但其本质是向远程服务器发送请求，所以同学们也可以使用 python 中的 `requests` 库，以发送网络请求的方式也可以实现云端大模型的调用。学有余力的同学可以自行探索一下～"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b064eb36",
   "metadata": {
    "id": "727569494DA94889B1C0F50BBDA2F683",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Step 1：获取 [硅基流动](https://cloud.siliconflow.cn/i/ybUFvmqK) API Key  \n",
    "\n",
    "访问硅基流动平台官网，注册账号，在个人中心找到你的 API Key（密钥） 和 API Base URL（接口地址）。请务必保管好你的 API Key，不要泄露。  \n",
    "\n",
    "创建好账户之后，即可获取到 API Key 和 API Base URL。  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e06cc56",
   "metadata": {
    "id": "F6328E28A5F440CDA63BB91B0950CD66",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### Step 2: 调用大模型  \n",
    "\n",
    "接下来，我们安装`openai`库，用于调用大模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed5d54a7",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "dc86f9d9",
   "metadata": {
    "collapsed": false,
    "id": "F21F6254338D44BE832F1F046C02B392",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [],
    "trusted": true
   },
   "outputs": [],
   "source": [
    "!pip install -q openai"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df4ce9b7",
   "metadata": {
    "id": "1F65D1675B8443CD94435E4F75FB3DC5",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "使用以下代码调用云端API大模型："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aba8a328",
   "metadata": {
    "collapsed": false,
    "id": "F1F8901D95A64457921E3CC9E116C952",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [],
    "trusted": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "你好！😊 有什么我可以帮您的吗？或者您想聊些什么呢？\n"
     ]
    }
   ],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "client = OpenAI(api_key=\"your api key\", \n",
    "                base_url=\"https://api.siliconflow.cn/v1\")\n",
    "response = client.chat.completions.create(\n",
    "    model=\"Qwen/Qwen3-8B\",\n",
    "    messages=[\n",
    "        {'role': 'user', 'content': \"你好！\"}\n",
    "    ],\n",
    "    max_tokens=1024,\n",
    "    temperature=0.7,\n",
    "    stream=False\n",
    ")\n",
    "print(response.choices[0].message.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17d3df78",
   "metadata": {
    "id": "57F39E12E05346D395E5F06D9436AA6C",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "- temperature（温度）  \n",
    "  - 控制输出的随机性/创造性  \n",
    "  - 取值范围 0-1，值越大随机性越强  \n",
    "  - 0表示始终选择最可能的词  \n",
    "\n",
    "- max_tokens（最大令牌数）  \n",
    "  - 限制模型回复的最大长度  \n",
    "  - 1 token 约等于 4 个字符  \n",
    "  - 需要根据具体任务设置合适的值  \n",
    "\n",
    "- stream  \n",
    "  - 是否流式输出  \n",
    "  - 取值范围 True/False  \n",
    "  - True 表示流式输出，模型会边生成边输出  \n",
    "  - False 表示一次性输出全部结果"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af1521f8",
   "metadata": {
    "id": "33E0A32A01D94BF3A2EFDB3900D60419",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "我们把以上代码封装为一个函数，可以接受用户输入，历史对话和系统提示词，最终返回模型生成的文本："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2842c799",
   "metadata": {
    "collapsed": false,
    "id": "244CA24E3E4C4621B3DC7120518E3D38",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [],
    "trusted": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "👋😊 你好呀！今天过得怎么样呀？有什么我可以帮你的吗？\n"
     ]
    }
   ],
   "source": [
    "def chat_with_model(user_input: str, history: list = None, temperature: float = 0.7, system_prompt: str = None) -> str:\n",
    "    \"\"\"\n",
    "    与大模型进行对话的函数\n",
    "    \n",
    "    Args:\n",
    "        user_input: 用户输入的文本\n",
    "        history: 历史对话记录列表\n",
    "        temperature: 温度参数，控制输出的随机性\n",
    "        system_prompt: 系统提示词\n",
    "    \n",
    "    Returns:\n",
    "        str: 模型返回的文本\n",
    "    \"\"\"\n",
    "\n",
    "    # 初始化 OpenAI 客户端\n",
    "    client = OpenAI(api_key=\"your api key\", \n",
    "                base_url=\"https://api.siliconflow.cn/v1\")\n",
    "\n",
    "    # 初始化历史记录\n",
    "    if history is None:\n",
    "        history = []\n",
    "        \n",
    "    # 构建消息列表\n",
    "    messages = []\n",
    "    \n",
    "    # 添加系统提示词(如果有)\n",
    "    if system_prompt:\n",
    "        messages.append({\"role\": \"system\", \"content\": system_prompt})\n",
    "    \n",
    "    # 添加历史对话\n",
    "    for msg in history:\n",
    "        messages.append(msg)\n",
    "    \n",
    "    # 添加当前用户输入\n",
    "    messages.append({\"role\": \"user\", \"content\": user_input})\n",
    "    \n",
    "    # 调用API获取响应\n",
    "    response = client.chat.completions.create(\n",
    "        model=\"Qwen/Qwen3-8B\",\n",
    "        messages=messages,\n",
    "        temperature=temperature\n",
    "    )\n",
    "    \n",
    "    # 返回模型回复的文本\n",
    "    return response.choices[0].message.content\n",
    "\n",
    "\n",
    "print(chat_with_model(\"你好哇\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef00e93e",
   "metadata": {
    "id": "B07A4D29F58745F1B658454556997D97",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 2.2 本地部署与调用：更强的掌控力  \n",
    "\n",
    "调用云端的大模型固然很方便，但它存在一些问题。比如：数据的隐私性。当同学们对外发送请求的时候，本地数据已经泄漏给外界了。所以当我们用一些特殊需求时，往往会采用本地部署大模型然后进行调用。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74765a0f",
   "metadata": {
    "id": "D4CDBD6EB6F34128AE021CAB730CF2F0",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 2.2.1 使用 `transformers` 部署大模型  \n",
    "\n",
    "`transformers` 是 Hugging Face 公司开发的一个库，是当下进行NLP（自然语言处理）和模型研究开发的最流行的一个python库。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ee5219c",
   "metadata": {
    "id": "50A45FE41A5143A0B0C31C16512B6A31",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "> 注：不知道大家有没有完成上一节布置的课后作业？（下载 Qwen3-4B 模型）  \n",
    "\n",
    "没有下载好的同学可以运行以下代码，在 ModelScope 平台下载模型。模型大小在 8G 左右。  \n",
    "\n",
    "> 注：如果模型下载的环境也没有配置完全，可以再阅读以下第一节的内容，进行环境配置。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e86f5435",
   "metadata": {
    "collapsed": false,
    "id": "523483A23B7B418594F527AF29CC9649",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [],
    "trusted": true
   },
   "outputs": [],
   "source": [
    "!pip install -q modelscope transformers accelerate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03c62155",
   "metadata": {
    "collapsed": false,
    "id": "B1526F10A0E84B2794798A48CC5F0708",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [],
    "trusted": true
   },
   "outputs": [],
   "source": [
    "from modelscope import snapshot_download\n",
    "\n",
    "model_dir = snapshot_download('Qwen/Qwen3-4B-Thinking-2507', cache_dir='/root/autodl-tmp/model', revision='master')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a943887",
   "metadata": {
    "id": "CC480C00F704410381BBD8C74F8F86F4",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "模型下载完成之后，就可以使用 `transformers` 库加载模型了，首先我们需要安装一些加载模型的环境配置，如果同学们在第一节安装过了，那就不需要再安装了。  \n",
    "\n",
    "```python  \n",
    "!pip install -q modelscope transformers accelerate  \n",
    "```\n",
    "\n",
    "> 注：没有配置环境的同学，请复制以上代码到自己的环境中运行安装。  \n",
    "\n",
    "配置好环境的同学，可以使用以下代码运行加载模型。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d2b87c4f",
   "metadata": {
    "collapsed": false,
    "id": "24094CB5D17C43D99736C218B3A1652D",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [],
    "trusted": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "Loading checkpoint shards: 100%|██████████| 3/3 [00:02<00:00,  1.22it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "thinking content: <think>\n",
      "嗯，用户让我介绍一下自己。首先，我需要确定用户的需求是什么。可能他们想了解我的功能、用途，或者想确认我是否适合他们的需求。用户可能是在开始对话前想先了解我的基本信息，或者他们可能对AI助手不太熟悉，需要一些引导。\n",
      "\n",
      "接下来，我要考虑如何组织回答。用户可能希望得到一个简洁明了的介绍，涵盖我的主要功能和应用场景。同时，我需要确保回答友好且易于理解，避免使用过于技术化的术语，让不同背景的用户都能明白。\n",
      "\n",
      "然后，我需要检查是否有遗漏的重要信息。比如，我的训练数据、支持的语言、是否支持多轮对话等。这些都是用户可能关心的点。另外，是否需要提到我的应用场景，比如回答问题、创作内容、提供建议等，这些都能帮助用户理解我的用途。\n",
      "\n",
      "还要注意用户可能的深层需求。他们可能想知道我是否能帮助他们解决问题，或者是否适合他们的特定需求。因此，在介绍中提到我的能力范围和适用场景会很有帮助。\n",
      "\n",
      "最后，确保回答结构清晰，分点说明，让用户一目了然。同时，保持语气友好，鼓励用户提出问题或进一步交流，以促进后续的互动。\n",
      "</think>\n",
      "content: 你好！我是通义千问，一个由阿里巴巴集团研发的大型语言模型，旨在帮助用户解答问题、创作内容、提供建议等。我的训练数据来自互联网文本，覆盖广泛的知识领域，能够进行多轮对话，支持多种语言（包括中文、英文、阿拉伯语、西班牙语等）。我专注于提供准确、有用的信息，并尽力满足用户的各种需求。如果你有任何问题或需要帮助，欢迎随时告诉我！\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
    "\n",
    "# 设置模型本地路径\n",
    "model_name = \"/root/autodl-tmp/model/Qwen/Qwen3-4B-Thinking-2507\"\n",
    "\n",
    "# 加载分词器和模型\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\n",
    "model = AutoModelForCausalLM.from_pretrained(\n",
    "    model_name,\n",
    "    torch_dtype=\"auto\",  # 自动选择合适的数据类型\n",
    "    device_map=\"cuda:0\",    # 自动选择可用设备(CPU/GPU)\n",
    "    trust_remote_code=True\n",
    ")\n",
    "\n",
    "# 准备模型输入\n",
    "prompt = \"你好，请介绍一下自己\"\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": prompt}\n",
    "]\n",
    "text = tokenizer.apply_chat_template(\n",
    "    messages,\n",
    "    tokenize=False,\n",
    "    add_generation_prompt=True,\n",
    "    enable_thinking=True # 选择是否打开深度推理模式\n",
    ")\n",
    "# 将输入文本转换为模型可处理的张量格式\n",
    "model_inputs = tokenizer([text], return_tensors=\"pt\").to(model.device)\n",
    "\n",
    "# 生成文本\n",
    "generated_ids = model.generate(\n",
    "    **model_inputs,\n",
    "    max_new_tokens=32768  # 设置最大生成token数量\n",
    ")\n",
    "# 提取新生成的token ID\n",
    "output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist() \n",
    "\n",
    "# 解析思考内容\n",
    "try:\n",
    "    # rindex finding 151668 (</think>)\n",
    "    # 查找结束标记\"</think>\"的位置\n",
    "    index = len(output_ids) - output_ids[::-1].index(151668)\n",
    "except ValueError:\n",
    "    index = 0\n",
    "\n",
    "# 解码思考内容和最终回答\n",
    "thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip(\"\\n\")\n",
    "content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip(\"\\n\")\n",
    "\n",
    "# 打印结果\n",
    "print(\"thinking content:\", thinking_content)\n",
    "print(\"content:\", content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0a2149f9",
   "metadata": {
    "collapsed": false,
    "id": "6A29A82449754FACAECAC5E4E4F220B9",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [],
    "trusted": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<|im_start|>user\n",
      "你好，请介绍一下自己<|im_end|>\n",
      "<|im_start|>assistant\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 准备模型输入\n",
    "prompt = \"你好，请介绍一下自己\"\n",
    "messages = [\n",
    "    {\"role\": \"user\", \"content\": prompt}\n",
    "]\n",
    "text = tokenizer.apply_chat_template(\n",
    "    messages,\n",
    "    tokenize=False,\n",
    "    add_generation_prompt=True,\n",
    "    enable_thinking=True # 选择是否打开深度推理模式\n",
    ")\n",
    "print(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8eab5665",
   "metadata": {
    "id": "719DA0300AFB41D788096029C641A95F",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "同样我们也可以将其封装为一个函数，如下代码所示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c5e8030",
   "metadata": {
    "collapsed": false,
    "id": "88282FA137624CB09AEF3E1CBFE58780",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [],
    "trusted": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.12/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "Loading checkpoint shards: 100%|██████████| 3/3 [00:02<00:00,  1.21it/s]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Qwen3ForCausalLM(\n",
       "  (model): Qwen3Model(\n",
       "    (embed_tokens): Embedding(151936, 2560)\n",
       "    (layers): ModuleList(\n",
       "      (0-35): 36 x Qwen3DecoderLayer(\n",
       "        (self_attn): Qwen3Attention(\n",
       "          (q_proj): Linear(in_features=2560, out_features=4096, bias=False)\n",
       "          (k_proj): Linear(in_features=2560, out_features=1024, bias=False)\n",
       "          (v_proj): Linear(in_features=2560, out_features=1024, bias=False)\n",
       "          (o_proj): Linear(in_features=4096, out_features=2560, bias=False)\n",
       "          (q_norm): Qwen3RMSNorm((128,), eps=1e-06)\n",
       "          (k_norm): Qwen3RMSNorm((128,), eps=1e-06)\n",
       "        )\n",
       "        (mlp): Qwen3MLP(\n",
       "          (gate_proj): Linear(in_features=2560, out_features=9728, bias=False)\n",
       "          (up_proj): Linear(in_features=2560, out_features=9728, bias=False)\n",
       "          (down_proj): Linear(in_features=9728, out_features=2560, bias=False)\n",
       "          (act_fn): SiLU()\n",
       "        )\n",
       "        (input_layernorm): Qwen3RMSNorm((2560,), eps=1e-06)\n",
       "        (post_attention_layernorm): Qwen3RMSNorm((2560,), eps=1e-06)\n",
       "      )\n",
       "    )\n",
       "    (norm): Qwen3RMSNorm((2560,), eps=1e-06)\n",
       "    (rotary_emb): Qwen3RotaryEmbedding()\n",
       "  )\n",
       "  (lm_head): Linear(in_features=2560, out_features=151936, bias=False)\n",
       ")"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from transformers import AutoModelForCausalLM, AutoTokenizer\n",
    "\n",
    "# 设置模型本地路径\n",
    "model_name = \"/root/autodl-tmp/model/Qwen/Qwen3-4B-Thinking-2507\"\n",
    "\n",
    "# 加载分词器和模型\n",
    "tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)\n",
    "model = AutoModelForCausalLM.from_pretrained(\n",
    "    model_name,\n",
    "    torch_dtype=\"auto\",  # 自动选择合适的数据类型\n",
    "    device_map=\"cuda:0\",    # 自动选择可用设备(CPU/GPU)\n",
    "    trust_remote_code=True\n",
    ")\n",
    "model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ce3d30c9",
   "metadata": {
    "collapsed": false,
    "id": "09E2CF4A3EAA433BA6651576143E0336",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [],
    "trusted": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "==================== 单轮对话结果 ====================\n",
      "thinking content: <think>\n",
      "好的，用户让我介绍一下自己。首先，我需要确定用户的需求是什么。可能他们想了解我的功能、用途，或者想确认我是否能帮助他们解决问题。用户可能刚接触AI助手，不太清楚我的具体能力，或者他们可能有特定的问题需要解决，想看看我是否适用。\n",
      "\n",
      "接下来，我要考虑如何组织回答。用户可能希望得到一个简洁明了的介绍，包括我的主要功能、应用场景以及如何帮助他们。需要避免使用过于技术化的术语，保持口语化，让用户容易理解。\n",
      "\n",
      "然后，我需要确保覆盖所有关键点：我的名称、身份、主要功能、应用场景、语言能力、学习能力、伦理原则等。同时，要突出我的优势，比如多语言支持、快速学习、伦理规范，这样用户能感受到我的可靠性和实用性。\n",
      "\n",
      "另外，用户可能有潜在的需求，比如他们可能在寻找一个可以处理复杂任务的助手，或者需要多语言支持。因此，在介绍中要提到这些方面，让用户知道我能够满足他们的多样化需求。\n",
      "\n",
      "还要注意回答的结构，先总述，再分点详细说明，最后邀请用户提问，这样既清晰又友好。同时，保持回答的积极态度，鼓励用户提出问题，促进进一步的互动。\n",
      "\n",
      "最后，检查是否有遗漏的重要信息，比如是否提到我的训练数据、更新频率，或者是否支持特定的行业应用。确保回答全面，但不过于冗长，让用户能快速抓住重点。\n",
      "</think>\n",
      "content: 你好！我是通义千问，是阿里巴巴集团旗下的通义实验室研发的AI助手。我具备强大的语言理解和生成能力，可以进行多轮对话、回答问题、创作文本、编程、逻辑推理等多种任务。我支持多种语言，包括中文、英文、法语、西班牙语等，并且能够根据用户的需求进行学习和优化。\n",
      "\n",
      "我的主要功能包括：\n",
      "1. **信息查询**：可以回答各种问题，提供知识和信息。\n",
      "2. **创作辅助**：帮助撰写文章、故事、邮件、剧本等。\n",
      "3. **编程支持**：可以编写和调试代码，解释编程概念。\n",
      "4. **多语言翻译**：支持多种语言的翻译和语言转换。\n",
      "5. **逻辑推理**：进行数学计算、逻辑分析和问题解决。\n",
      "\n",
      "我能够根据用户的需求进行快速学习和适应，同时遵循严格的伦理原则，确保对话的安全性和合规性。如果你有任何问题或需要帮助，欢迎随时告诉我！\n",
      "\n",
      "==================== 多轮对话结果 ====================\n",
      "thinking content: <think>\n",
      "好的，用户让我帮忙写一首诗。首先，我需要确定用户的需求是什么。他们可能想要一首有特定主题的诗，或者只是随便写一首。考虑到用户没有给出具体的方向，我应该先询问他们想要的风格、主题或者情感基调。比如，是现代诗还是古体诗？是表达爱情、自然，还是其他情感？\n",
      "\n",
      "然后，我需要考虑用户可能的深层需求。也许他们正在准备某个活动，比如生日、纪念日，或者只是想练习写诗。也有可能他们需要一首诗来表达某种情绪，比如孤独、希望或者悲伤。这时候，我应该提供一些选项，让用户更容易选择。\n",
      "\n",
      "另外，用户可能对诗歌的结构和用词有特定的要求。比如，是否需要押韵，是否有特定的意象或比喻。我需要确保在写诗时，能够满足这些潜在的需求，同时保持诗歌的连贯性和美感。\n",
      "\n",
      "接下来，我应该用友好的语气邀请用户提供更多细节，这样可以确保生成的诗更符合他们的期望。同时，保持回应的简洁和自然，避免让用户感到压力。比如，可以问：“当然可以！你希望这首诗表达什么样的情感或主题呢？是爱情、自然，还是其他？另外，你有偏好的诗体风格吗？”这样既明确又不会让用户觉得过于繁琐。\n",
      "\n",
      "最后，我需要确保自己能够灵活应对不同的请求，无论是现代诗还是古体诗，都能写出有感染力的作品。同时，保持耐心和积极的态度，让用户感到被重视和支持。\n",
      "</think>\n",
      "content: 当然可以！你希望这首诗表达什么样的情感或主题呢？是爱情、自然，还是其他？另外，你有偏好的诗体风格吗？比如现代诗、古体诗，或者有特定的意象或比喻？我可以根据你的想法来创作。\n"
     ]
    }
   ],
   "source": [
    "def chat_with_local_model(model, tokenizer, user_input: str, history: list = None, temperature: float = 0.7, max_new_tokens: int = 1024, enable_thinking: bool = True) -> dict:\n",
    "    \"\"\"\n",
    "    使用本地大模型进行对话的函数\n",
    "    \n",
    "    Args:\n",
    "        model: 加载好的模型\n",
    "        tokenizer: 加载好的分词器\n",
    "        history: 历史对话记录列表，格式为 [{\"role\": \"user\", \"content\": \"...\"}, {\"role\": \"assistant\", \"content\": \"...\"}]\n",
    "        temperature: 温度参数，控制输出的随机性 (0.0-1.0)\n",
    "        max_new_tokens: 最大生成token数量\n",
    "        enable_thinking: 是否启用深度推理模式\n",
    "    \n",
    "    Returns:\n",
    "        dict: 包含思考内容和最终回答的字典 {\"thinking_content\": str, \"content\": str}\n",
    "    \"\"\"\n",
    "    \n",
    "    # 初始化历史记录\n",
    "    if history is None:\n",
    "        history = []\n",
    "    \n",
    "    # 构建消息列表\n",
    "    messages = []\n",
    "    \n",
    "    # 添加历史对话\n",
    "    for msg in history:\n",
    "        messages.append(msg)\n",
    "    \n",
    "    # 添加当前用户输入\n",
    "    messages.append({\"role\": \"user\", \"content\": user_input})\n",
    "    \n",
    "    # 应用聊天模板\n",
    "    text = tokenizer.apply_chat_template(\n",
    "        messages,\n",
    "        tokenize=False,\n",
    "        add_generation_prompt=True,\n",
    "        enable_thinking=enable_thinking  # 选择是否打开深度推理模式\n",
    "    )\n",
    "    \n",
    "    # 将输入文本转换为模型可处理的张量格式\n",
    "    model_inputs = tokenizer([text], return_tensors=\"pt\").to(model.device)\n",
    "    \n",
    "    # 生成文本\n",
    "    generated_ids = model.generate(\n",
    "        **model_inputs,\n",
    "        max_new_tokens=max_new_tokens,  # 设置最大生成token数量\n",
    "        temperature=temperature,\n",
    "        do_sample=True if temperature > 0 else False  # 当temperature > 0时启用采样\n",
    "    )\n",
    "    \n",
    "    # 提取新生成的token ID\n",
    "    output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()\n",
    "    \n",
    "    # 解析思考内容\n",
    "    try:\n",
    "        # rindex finding 151668 (</think>)\n",
    "        # 查找结束标记\"</think>\"的位置\n",
    "        index = len(output_ids) - output_ids[::-1].index(151668)\n",
    "    except ValueError:\n",
    "        index = 0\n",
    "    \n",
    "    # 解码思考内容和最终回答\n",
    "    thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip(\"\\n\")\n",
    "    content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip(\"\\n\")\n",
    "    \n",
    "    return {\n",
    "        \"thinking_content\": thinking_content,\n",
    "        \"content\": content\n",
    "    }\n",
    "\n",
    "# 使用示例\n",
    "if __name__ == \"__main__\":\n",
    "    \n",
    "    # 单轮对话示例\n",
    "    result = chat_with_local_model(\n",
    "        model=model,\n",
    "        tokenizer=tokenizer,\n",
    "        user_input=\"你好，请介绍一下自己\",\n",
    "        temperature=0.7\n",
    "    )\n",
    "    \n",
    "    print(\"\\n==================== 单轮对话结果 ====================\")\n",
    "    print(\"thinking content:\", result[\"thinking_content\"])\n",
    "    print(\"content:\", result[\"content\"])\n",
    "    \n",
    "    # 多轮对话示例\n",
    "    history = [\n",
    "        {\"role\": \"user\", \"content\": \"你好\"},\n",
    "        {\"role\": \"assistant\", \"content\": \"你好！我是Qwen，一个AI助手。\"}\n",
    "    ]\n",
    "    \n",
    "    result = chat_with_local_model(\n",
    "        model=model,\n",
    "        tokenizer=tokenizer,\n",
    "        user_input=\"你能帮我写一首诗吗？\",\n",
    "        history=history,\n",
    "        temperature=0.8\n",
    "    )\n",
    "    \n",
    "    print(\"\\n==================== 多轮对话结果 ====================\")\n",
    "    print(\"thinking content:\", result[\"thinking_content\"])\n",
    "    print(\"content:\", result[\"content\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41f76073",
   "metadata": {
    "id": "380B53E628394FD99FA7B78AB7566B9D",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "虽然使用 transformers 在本地部署模型能让我们获得完整的控制权限，但这种方式存在一定的性能瓶颈，尤其在首次推理时表现明显。这种本地部署方式更适合进行简单模型加载测试或算法研究工作，但若要将模型打造成一个支持高并发调用的服务，其计算效率则显得捉襟见肘。这便是 transformers 本地部署方案在性能方面的主要局限。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a153a86",
   "metadata": {
    "id": "593B8B24A3B540AF9383F10263C611FB",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "### 2.2.2 进阶：使用 vLLM 进行高性能部署"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d098504",
   "metadata": {
    "id": "A621795C1A92471598327E2E9A7AF40A",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "为了解决上述效率问题，我们可以使用像 vLLM 这样的高性能推理框架。vLLM 通过 `PagedAttention` 等先进的内存管理技术，极大地提升了模型的吞吐量（每秒能处理的请求数）和响应速度，特别是在并发请求的场景下。  \n",
    "\n",
    "首先第一步，安装 vllm，直接安装即可～  \n",
    "\n",
    "```python  \n",
    "!pip install vllm  \n",
    "```\n",
    "\n",
    "> 注意：安装时间较长，请耐心等待。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "95c977da",
   "metadata": {
    "collapsed": false,
    "id": "92107BA39255451C94CD9235E1534C23",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [],
    "trusted": true
   },
   "outputs": [],
   "source": [
    "!pip install -q vllm "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1fb501d",
   "metadata": {
    "id": "C09B5EF06F9048028309D0C69F6CD465",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "使用以下命令在命令行终端启动 vllm 服务：  \n",
    "\n",
    "```bash  \n",
    "vllm serve /root/autodl-tmp/model/Qwen/Qwen3-4B-Thinking-2507 \\\n",
    "    --served-model-name Qwen3-4B \\\n",
    "    --max_model_len 1024 \\\n",
    "    --reasoning-parser deepseek_r1 \n",
    "```\n",
    "\n",
    "以上命令中各参数的含义如下：  \n",
    "\n",
    "- `/root/autodl-tmp/model/Qwen/Qwen3-4B-Thinking-2507`: 模型路径，指向本地存储的 Qwen 3.0 6B 模型文件  \n",
    "- `--served-model-name Qwen3-4B`: 服务启动后使用的模型名称  \n",
    "- `--max_model_len 1024`: 设置模型最大处理的序列长度为 1024 个 token  \n",
    "- `--reasoning-parser deepseek_r1`: 使用 deepseek_r1 作为推理解析器  \n",
    "\n",
    "这些参数配置了 vllm 服务的基本运行参数，包括模型位置、服务名称、序列长度限制以及推理相关的功能设置。  \n",
    "\n",
    "> 注意：如果想要对 vllm 服务进行更多定制化配置，建议参考 vllm 官方文档。  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4292f1d",
   "metadata": {
    "id": "A657826ECE1E4799AEA9BD6348843D1D",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "然后就可以像使用 2.1 云端大模型一样的方式来调用 vllm 启动的模型服务，如下代码所示："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a8f223b",
   "metadata": {
    "collapsed": false,
    "id": "2E9A6AAB9B2E463E98CB5E99E57E92FE",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": [],
    "trusted": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "你好！很高兴见到你，有什么我可以帮你的吗？😊\n"
     ]
    }
   ],
   "source": [
    "from openai import OpenAI\n",
    "\n",
    "client = OpenAI(api_key=\"xxx\", \n",
    "                base_url=\"http://127.0.0.1:8000/v1\")\n",
    "response = client.chat.completions.create(\n",
    "    model=\"Qwen3-4B\",\n",
    "    messages=[\n",
    "        {'role': 'user', 'content': \"你好哇\"}\n",
    "    ],\n",
    "    max_tokens=512,\n",
    "    temperature=0.7,\n",
    "    stream=False\n",
    ")\n",
    "print(response.choices[0].message)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "652056ef",
   "metadata": {
    "id": "EC80538C568D41959570F1F3733DB9BC",
    "jupyter": {},
    "notebookId": "68787ddac2ed7c11752d125c",
    "runtime": {
     "execution_status": null,
     "is_visible": false,
     "status": "default"
    },
    "scrolled": false,
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## 课后作业  \n",
    "\n",
    "请同学们尝试使用 python 中的 requests 来调用模型服务。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d612110e",
   "metadata": {},
   "source": [
    "课后作业代码参考：\n",
    "\n",
    "```python\n",
    "import requests\n",
    "\n",
    "url = \"http://127.0.0.1:8000/v1/chat/completions\"\n",
    "\n",
    "payload = {\n",
    "    \"model\": \"Qwen3-4B\",\n",
    "    \"messages\": [\n",
    "        {\n",
    "            \"role\": \"user\",\n",
    "            \"content\": \"请详细介绍一下你自己～\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "headers = {\n",
    "    \"Authorization\": \"Bearer <token>\",\n",
    "    \"Content-Type\": \"application/json\"\n",
    "}\n",
    "\n",
    "response = requests.post(url, json=payload, headers=headers)\n",
    "\n",
    "print(response.json())\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e21c4642",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "nlp",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
